Skip to content

Reduce GCS QPS on ArrayRecordReader open via ReadLast. - #212

Open
copybara-service[bot] wants to merge 1 commit into
mainfrom
test_992624347
Open

copybara-service[bot] wants to merge 1 commit into
mainfrom
test_992624347

Conversation

@copybara-service

@copybara-service copybara-service Bot commented Oct 3, 2026 •

Copy link
Copy Markdown

Reduce GCS QPS on ArrayRecordReader open via ReadLast.

Previously, opening a gs:// ArrayRecord shard issued 3 separate GCS HTTP GET
requests (5 when index_storage_option is offloaded):

  1. An unbounded GET at offset 0 during GcsReader construction solely to obtain
    the object size (and then aborted mid-stream).
  2. A GET via reader->NewReader(size - 64KB) to read the Riegeli postscript.
  3. A GET via reader->NewReader(footer_offset) to read the ArrayRecord footer
    (and additional GETs in OffloadedChunkOffset::operator[] when
    index_storage_option is offloaded).

Open riegeli::GcsReader with google::cloud::storage::ReadLast(1 MiB) (using
a per-reader google::cloud::storage::Client instance to avoid cross-thread
libcurl handle pool contention and Curl_peer refcount data races) so the
initial HTTP GET retrieves the total object size (Content-Range) and
prefetches the last 1 MiB of the shard. In ArrayRecordReaderBase::Initialize(),
pull the prefetched tail into reader's buffer (resetting the cursor to
limit() so available() == 0 on close) and read the postscript and footer
(including offloaded chunk index lookups in OffloadedChunkOffset::operator[])
directly from reader's buffered tail via ReadChunk when
footer_offset >= reader->start_pos(), without spawning subordinate
NewReader HTTP streams.

Real-GCS integration benchmarks show 3.0×–5.0× fewer HTTP GETs on open (23.0× fewer on shards < 1 MiB), 2.0×–3.0× fewer TLS handshakes on open, 2.2×–4.7× faster shard initialization, and 1.13×–1.65× higher bulk read throughput with zero data races.

@copybara-service
copybara-service Bot force-pushed the test_992624347 branch 4 times, most recently from 1c4bdee to 88b11c1 Compare October 5, 2026 20:10
Previously, opening a `gs://` ArrayRecord shard issued 3 separate GCS HTTP GET
requests (5 when `index_storage_option` is `offloaded`):
1. An unbounded GET at offset 0 during `GcsReader` construction solely to obtain
   the object size (and then aborted mid-stream).
2. A GET via `reader->NewReader(size - 64KB)` to read the Riegeli postscript.
3. A GET via `reader->NewReader(footer_offset)` to read the ArrayRecord footer
   (and additional GETs in `OffloadedChunkOffset::operator[]` when
   `index_storage_option` is `offloaded`).

Open `riegeli::GcsReader` with `google::cloud::storage::ReadLast(1 MiB)` (using
a per-reader `google::cloud::storage::Client` instance to avoid cross-thread
`libcurl` handle pool contention and `Curl_peer` refcount data races) so the
initial HTTP GET retrieves the total object size (`Content-Range`) and
prefetches the last 1 MiB of the shard. In `ArrayRecordReaderBase::Initialize()`,
pull the prefetched tail into `reader`'s buffer (resetting the cursor to
`limit()` so `available() == 0` on close) and read the postscript and footer
(including offloaded chunk index lookups in `OffloadedChunkOffset::operator[]`)
directly from `reader`'s buffered tail via `ReadChunk` when
`footer_offset >= reader->start_pos()`, without spawning subordinate
`NewReader` HTTP streams.

Real-GCS integration benchmarks show 3.0×–5.0× fewer HTTP GETs on open (23.0× fewer on shards < 1 MiB), 2.0×–3.0× fewer TLS handshakes on open, 2.2×–4.7× faster shard initialization, and 1.13×–1.65× higher bulk read throughput with zero data races.

PiperOrigin-RevId: 992624347

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant